chore: forward-port clean v5-next backlog to next#24849
Open
PhilWindle wants to merge 35 commits into
Open
Conversation
PhilWindle
force-pushed
the
port-v5-backlog-to-next
branch
from
July 21, 2026 16:17
7eea866 to
66aad56
Compare
…-account (#24476) ## Summary Wires a real `--funding-account` option into `aztec validator-keys new` (previously commented out with a TODO, so operators had to hand-edit the keystore JSON), and adds a dedicated `set-funding-account` subcommand for existing keystores. Per review feedback, `add` does **not** take `--funding-account`: the funding account is a keystore-level field (the only one `KeystoreManager.createFundingSigner` reads), so setting it from `add` would be a global mutation disguised as a per-validator flag. Use `set-funding-account` instead: ``` aztec validator-keys set-funding-account <keystore.json> <privateKey|address> [--remote-signer <url>] [--password <str>] ``` ## Behavior - The funding account value accepts either a 32-byte private key or a 20-byte address. - An address requires `--remote-signer` (a local funder needs its private key to sign funding txs); it is stored as `{ address, remoteSignerUrl }`. - With `--password`, a plaintext funder key is encrypted to a JSON V3 file and replaced with a `{ path, password }` reference, mirroring how attester/publisher keys are handled. - The value is written at the keystore top level (`keystore.fundingAccount`). Validator-level funding exists in the schema but is never consumed at runtime, so it is intentionally not emitted. - `set-funding-account` replaces an existing funding account with a warning. ## Changes - `index.ts` — real `--funding-account` option on `new`; new `set-funding-account` subcommand. - `set_funding_account.ts` — new command: validate, resolve, optionally encrypt, write `keystore.fundingAccount`. - `utils.ts` — `validateFundingAccountOptions` (normalize + validate key/address, require remote-signer for address form). - `shared.ts` — `resolveFundingAccount` and `encryptFundingAccountToFile`; extracted the ETH JSON V3 encryption helper for reuse. - `new.ts` — validate, resolve, optionally encrypt, set `keystore.fundingAccount`. - `valkeys.test.ts` — 17 new tests covering validation, private-key/address/password paths on `new`, and set/replace on `set-funding-account`. ## Testing - `yarn workspace @aztec/cli test src/cmds/validator_keys/valkeys.test.ts` — 60 passed. - `yarn build`, `yarn format cli`, `yarn lint cli` — all clean. (cherry picked from commit 3aa68f6)
Reduce the mainnet SLASH_INACTIVITY_TARGET_PERCENTAGE default from 0.8 (80%) to 0.7 (70%) in spartan/environments/network-defaults.yml. Only the networks.mainnet preset is changed; devnet/testnet and the top-level slasher anchor (all 0.9) are untouched. (cherry picked from commit 71f8e3d)
Updates every README in the e2e test section to describe the suite as it will look once the nine open round-2 consolidation PRs land. This is docs-only — no code, config, or `bootstrap.sh` changes. ## What changed per README - **`end-to-end/README.md`** (root): repointed the `LOG_LEVEL` example off the deleted `proving/empty_blocks.test.ts` to `proving/default_node.test.ts`; linked the new p2p README from the category table (the p2p row now reads "yes" and points at `src/p2p/README.md`); and renamed the `setupBlockProducer` submission-window literal `1024` to the named constant `NO_REORG_SUBMISSION_EPOCHS`. - **`end-to-end/src/single-node/README.md`**: rewrote the `cross-chain/` row for the merged layout (`CrossChainMessagingTest` + `message_test_helpers.ts`, and the `l1_to_l2` / `l1_to_l2_inbox_drift` / `l2_to_l1` / `token_bridge` files); folded `public_payments`/`sponsored_payments` into `private_payments.parallel` and fixed the stale `failures.parallel` name to `failures`; replaced `slasher_config`/`sequencer_config` with the merged `runtime_config`; replaced `world_state_pruning`/`empty_blocks` with the merged `default_node`; added the previously-missing `snapshot_sync` to the `sync/` row; and applied the `NO_REORG_SUBMISSION_EPOCHS` rename. - **`end-to-end/src/multi-node/README.md`**: named `equivocation_offenses` in the slashing row (in place of the separate `duplicate_proposal`/`duplicate_attestation` files) and applied the `NO_REORG_SUBMISSION_EPOCHS` rename in the `MOCK_GOSSIP_MULTI_VALIDATOR_OPTS` preset. - **`end-to-end/src/p2p/README.md`** (new): created with the content added by #24500. - **`end-to-end/src/automine/README.md`** and **`end-to-end/src/infra/README.md`**: untouched — both are behavior-level (no file names) or cover a directory none of the nine PRs change, so they remain accurate as-landed. --------- Co-authored-by: AztecBot <tech@aztec-labs.com> (cherry picked from commit 1cc5f91)
## What / why
Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.
Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.
## Commit
- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
`failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.
## Diagnosis
The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.
Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:
- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
- bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)
**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.
No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.
## Expected effect
No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.
Measured numbers from a full green CI run will be appended below once
available.
## Measured impact
Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.
Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):
- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
**100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
hook accounting.
Fees suites (`private_payments.parallel` × 8 shards + `failures`):
- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
`warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
≈ −1s, i.e. fully attributed.
Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.
Fixes A-1407
(cherry picked from commit 1476179)
…24566) PR 4 of the round-4 e2e speedup effort: extend the merged `warpWithSequencersPaused` primitive (from #24475, proven on `pipeline_prune`) to the remaining production-sequencer-bound dead waits. One commit per converted site so a CI-hostile one reverts in isolation. On the fresh `merge-train/spartan-v5` base, most of the sites the plan targeted have already been warp-optimized (they use `warpToBuildWindowForSlot` / `warpToEpochStart` / `waitUntilNextEpochStarts`, and `pipeline_prune` already uses `warpWithSequencersPaused`). The residual waits in the remaining sites ride real-time consensus/production that a warp cannot safely collapse — warping would skip the checkpoints/attestations/offenses the tests are verifying. After reading each file on the fresh base, only one genuinely-dead wall-clock wait was left to convert; the rest are honest skips (per the round-3 skip rule: an honest skip beats a flaky convert). ## Converted (one commit) - **`multi-node/recovery/proposal_failure_recovery` — missed-L1-publish test.** Between "proposed chain reached slotTwo" and the prune, the test waited in wall-clock (`waitUntilL1Timestamp`) for the L1 clock to roll past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo blocks. The pipelined slotTwo broadcast has already reached every node and slotThree does not build until slotTwo, so nothing must be produced in that gap — a genuine dead wait, the direct `pipeline_prune` analog. Replaced with `warpWithSequencersPaused(..., { restart: false })`: pause across the warp (warping under a running sequencer would interrupt in-flight builds), keep the sequencers stopped until the prune is confirmed so no proposer builds on the unpruned tip, then restart them for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this suite's cadence) per run of that test. ## Skipped, with reasons - **`proposal_failure_recovery` — orphan-prune test (second `it`).** The prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the same slot (pipelining builds S2 during S1 while the orphan is pruned). Pausing the sequencers to warp would stop the S2 rebuild the test then asserts on. No pausable dead gap. - **`multi-node/recovery/equivocation_recovery`.** The dominant waits are the heal phase (`waitUntilCheckpointNumber` / `waitForAllNodesToReachCheckpoint` for baseline+2) which requires B/C to actually produce two checkpoints after A is stopped, plus `waitForOffenseOnNodes` which requires real slashing-round detection. Warping skips the very production being verified. - **`multi-node/slashing/slash_veto_demo`.** `waitForSubmittableRound` accrues inactivity offenses to quorum over proven blocks, and the veto/expiry logic rides real slashing-round consensus. This is proof-submission/consensus real time (the prover must actually run) — warp does not help, same as the fees `catchUpProvenChain` finding from round 3. - **`multi-node/slashing/attested_invalid_proposal`** (now under `slashing/`, not `invalid-attestations/` — base drift). Epoch advances already use cheatcode warps (`advanceToEpoch` / `advanceToNextEpoch`); the remaining waits are real-time committee/attestation/offense-detection rounds (bad proposer must build and broadcast, lazy validator must attest, honest node must detect the offense). - **`multi-node/block-production/multi_validator_node`.** Cost is real-time attestation + checkpoint building via an actual deploy tx; the only clock jump (`advanceToEpoch` past the validator-set lag) is already a cheatcode warp, and this test was just flake-fixed (#24543) so it is left untouched. - **`single-node/proving/optimistic.parallel` (`wait:epoch` tails, pool 2).** Already warp-optimized on this base: every epoch-boundary skip uses `warpToEpochStart` / `waitUntilNextEpochStarts` / `warpToBuildWindowForSlot`. The residual `wait:epoch` is the deliberate ~2-slot real-time tail (the epoch's final checkpoint must land and the sequencer must settle) — pausing across it would suppress that checkpoint. The `waitUntilCheckpointNumber(midCheckpoint)` waits require the sequencer to produce two mid-epoch checkpoints, and `waitUntilProvenCheckpointNumber(..., 240)` rides real prover latency. Nothing left to safely convert. - **`wait:tx-mined` sites (pool 3).** These are inclusion waits for in-flight builds, not waits for a future slot, so they do not qualify (as anticipated in the plan). Not converted. ## Local verification - `yarn build` (full TS project): passes. - `yarn format end-to-end` / `yarn lint end-to-end`: clean. - The converted test is a 4-validator mock-gossip pipelining suite, too heavy to run on the dev machine, so it was not executed locally. CI (`ci-no-fail-fast`) is the verifier; per-commit isolation contains any flake. ## Measured impact The conversion landed as scoped: `warp:sequencers-paused` goes from 1 occurrence run-wide (pipeline_prune, pre-existing from #24475) to 2 — the new one on the converted `proposal_failure_recovery` test. - Converted test ("prune and recover when proposer fails to publish to L1"): total 130.8s → 122.5s (−8.3s), body 114.6s → 109.2s (−5.4s). The added `warp:sequencers-paused` span is 36.1s. - The net saving is modest: the pause+warp+drain+deferred-restart cycle itself costs ~36s, only ~8s less than the real-time prune-boundary wait it replaced — not a full slot. The clean attribution is the span count (1→2); the ~8s test-total delta sits near the multi-node noise floor. - Untouched sites are flat within noise: the sibling orphan-prune test 93.7s → 91.5s, and pipeline_prune 260.4s → 258.7s. This is consistent with the PR's own finding that the warp lever is largely exhausted on this base — one convertible site, a single-test gain. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783387686257121 (x-fast). ## Notes for the rest of round 4 - No new shared helper was added — the converted site reuses the existing `warpWithSequencersPaused` on `SingleNodeTestContext` (inherited by `MultiNodeTestContext`). PRs 1a/1b/6/5/7 have nothing new to reuse from here. - Finding relevant to the round's premise: on this drifted base the multi-node recovery/slashing suites' large `wait:checkpoint`/`wait:epoch` pools are dominated by real production (checkpoint building, slashing rounds, prover latency), not dead clock time. The warp lever is largely exhausted there; PR 5 (shrink slot times) is the remaining cadence lever for those suites. Fixes A-1183 (cherry picked from commit cfb6c7f)
## What
PR 5 of the round-4 e2e speedup effort: shrink e2e slot times, the
highest-leverage remaining lever
now that the warp lever is exhausted (PR 4) and the fees setup costs are
pure inclusion latency at
CI cadence (PR 6). Every cadence-bound cost — setup txs, inclusion
waits, epoch walks — scales
linearly with what these commits cut.
Config-only. Three commits, in **ascending risk order** so the
CI-survey/fix phase can revert from
the top. A fourth candidate (WIDE_SLOT 72s → 48s) was investigated and
**dropped** — see below.
All guard math is recomputed from `stdlib/src/timetable/budgets.ts` +
`stdlib/src/timetable/proposer_timetable.ts`:
maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) must be
>= 1
where `S` = aztecSlotDuration, `init` = checkpointProposalInitTime (1s,
never clamped), `D` =
blockDurationMs/1000, `P` = p2pPropagationTime, `prepCp` =
checkpointProposalPrepareTime. Fast-profile
clamping (`P -> 0.5`, `prepCp -> 0.5`, `minBlock -> 1`) applies only
when `ethereumSlotDuration < 8`
(`FAST_PROFILE_ETHEREUM_SLOT_DURATION`). At or above 8s the production
budgets apply verbatim.
## Commit 1 (lowest risk) — `perf(e2e): cut default CI L1 block time to
8s`
`DEFAULT_L1_BLOCK_TIME`: `process.env.CI ? 12 : 8` → `8`. Local dev
already ran at 8, so this only
removes a CI-vs-local cadence asymmetry. Default single-node L2 slot
goes 24s → 16s.
Guard (default single-node: D = DEFAULT_BLOCK_DURATION_MS/1000 = 3,
production budgets P=2,
prepCp=1, init=1):
- before (eth 12): S = 2x12 = 24 → floor((24 - 1 - 3 - 4 - 1)/3) =
floor(15/3) = **5** blocks/cp
- after (eth 8): S = 2x8 = 16 → floor((16 - 1 - 3 - 4 - 1)/3) =
floor(7/3) = **2** blocks/cp
Both >= 1. eth=8 is AT the fast-profile boundary, so budgets stay at
production values (no clamping).
**Blast radius is much smaller than the round-4 plan assumed.** The plan
expected this to cut the
fees / cross-chain / bot families, but the base has drifted: those
suites now run on
`PIPELINING_SETUP_OPTS` / `AUTOMINE_E2E_OPTS` (both set
`ethereumSlotDuration: 4` explicitly) and no
longer read `DEFAULT_L1_BLOCK_TIME`. The genuine default-cadence
consumers (eth from
`SingleNodeTestContext.getSlotDurations`, no explicit eth) are four
single-node proving/recovery
suites: `proving/default_node`, `proving/cross_chain_public_message`,
`recovery/sync_after_reorg`,
`partial-proofs/multi_root`. The only other `process.env.CI` timing
coupling
(`multi-node/slashing/inactivity_setup.ts`, `process.env.CI ? 8 : 4`)
sets its own eth slot and does
not read `DEFAULT_L1_BLOCK_TIME`, so it is untouched.
## Commit 2 (medium risk) — `perf(e2e): cut reorg cadence to 24s slots`
`REORG_TIMING_BASE`: aztecSlotDuration 36 → 24, blockDurationMs 8000 →
5000 (epoch stays 4 slots).
Shared by both reorg profiles, both below the fast-profile boundary:
- `FAST_REORG_TIMING` (eth 4): proving/optimistic reorg describes,
l1-reorgs/
- `MULTI_VALIDATOR_REORG_TIMING` (eth 6, attestationPropagationTime
0.5): recovery/proposal_failure_recovery,
recovery/equivocation_recovery, high-availability/ha_sync,
high-availability/ha_checkpoint_handoff
Both run fast-profile: P = 0.5, prepCp = 0.5, init = 1.
Guard, FAST_REORG_TIMING and MULTI_VALIDATOR_REORG_TIMING (identical
budgets):
- before: S = 36, D = 8 → floor((36 - 1 - 8 - 1 - 0.5)/8) =
floor(25.5/8) = **3** blocks/cp
- after: S = 24, D = 5 → floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5)
= **3** blocks/cp
blockDurationMs drops 8000 → 5000 specifically to preserve 3
blocks/checkpoint. Leaving it at 8000
would give floor(13.5/8) = **1**, which would break the l1-reorgs
suites' `assertMultipleBlocksPerSlot(2)`
(they push TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block
checkpoints). All other timing scales
off `test.constants.slotDuration` (waits, warps).
equivocation_recovery's manual fit calc still holds
(3 x 5s = 15 + 0.5 init + 5 final block + 2.5 finalization = 23s <=
24s); its comment is updated.
## Commit 3 (higher risk) — `perf(e2e): cut multi-validator
block-production cadence to 24s slots`
`MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING`: aztecSlotDurationInL1Slots 3
→ 2 (36s → 24s at eth 12),
blockDurationMs 6000 → 4000. eth stays 12 (production budgets).
Consumers: block-production/simple,
first_slot, proof_boundary.
Guard (eth 12, production budgets prepCp=1, init=1; P = per-test
attestationPropagationTime), S=24, D=4:
- simple (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) =
**3** (was 4 at 36/6)
- proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = **3**
(was 4)
- first_slot (P=0.5): floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) =
**4** (was 4)
All >= 1. simple asserts no block-count (only no-sequencer-failures);
proof_boundary asserts
proof-vs-boundary timing that scales off `test.constants`; first_slot
keeps 4 blocks/checkpoint
(its comment stays accurate). So the drop 4 → 3 is harmless for those
three.
`high_tps` is **pinned to the old 36s/6s cadence at its call site**
(`aztecSlotDurationInL1Slots: 3`,
`blockDurationMs: 6000` in its setupOpts, overriding the shared
profile). Its checkpoint packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block
cannot hold it, and the suite
hard-asserts `max-checkpoint-length == 4` and `max-txs-per-block == 2`.
Pinning keeps high_tps at its
tuned cadence rather than rewriting its whole timing model. The other
three suites still take the cut.
## Dropped candidate — WIDE_SLOT 72s → 48s
Investigated per the plan's stretch item and **dropped**.
`blob_promotion` hard-asserts a checkpoint
with >= `PIPELINE_EXPECTED_BLOCKS_PER_CHECKPOINT = 8` blocks. At the
profile's D = 5.5s:
- current 72/12: floor((72 - 1 - 5.5 - 4 - 1)/5.5) = floor(60.5/5.5) =
**11** blocks/cp (fits 8)
- 48/12: floor((48 - 1 - 5.5 - 4 - 1)/5.5) = floor(36.5/5.5) = **6**
blocks/cp (cannot fit 8)
Forcing 8 blocks would require shrinking D to ~4.5s, which lands the
guard at exactly 8 with zero
margin — under the deliberate `mockGossipSubNetworkLatency: 500ms` the
suite runs with, and on top of
the profile's documented A-914 constraint (pipelined
multiple-blocks-per-slot starves non-proposer
nodes with `CheckpointNumberNotSequentialError` when the L2 slot is
tightened, worsened by a shorter
slot). This is the "provably can't fit" case, and it cannot be verified
locally (heavy multi-node
pipelining), so it is not shipped. proposed_chain / cross_chain_messages
(only need 2 blocks/cp) would
survive the guard but share the A-914 risk; not worth the exposure for
one profile.
## Local verification
- **Commit 1: verified.** Ran `single-node/proving/default_node.test.ts
-t 'returns initial block data'`
locally (eth=8 = the new CI cadence, which local already uses). The node
came up at
`ethereumSlotDuration: 8, aztecSlotDuration: 16, blockDurationMs: 3000`
and logged
`Sequencer timetable initialized with 2 blocks per slot
{"maxNumberOfBlocks":2}` — exactly the guard
value above. No "Invalid timing configuration", no missed-slot warnings,
test passed. Budgets stayed
at production values (`attestationPropagationTime: 2, minBlockDuration:
2`), confirming eth=8 does not
trip fast-profile clamping.
- **Commits 2 and 3: guard math + CI only.** The reorg and
multi-validator block-production suites are
too heavy for the dev machine (multi-node + real-time waits + proving).
The guard math above is the
verification; CI validates end-to-end. The formula was confirmed exact
against commit 1's live run.
## Revert protocol for the fix phase
Commits are ordered by ascending risk; revert individual commits from
the top on flake rather than
fighting them:
1. **Any block-production suite flake** (simple / first_slot /
proof_boundary) → revert commit 3 first.
proof_boundary is the most real-time-sensitive (it warps to N-3 then
runs the boundary in real time
at the tighter 24s cadence); watch it first.
2. **Any reorg / prune / HA flake** (l1-reorgs, proving/optimistic reorg
cases,
recovery/proposal_failure_recovery, recovery/equivocation_recovery,
high-availability/*) → revert
commit 2. Watch the l1-reorgs `assertMultipleBlocksPerSlot(2)`
assertions and equivocation_recovery's
slashing-round timing.
3. **Any of the four default-cadence single-node proving/recovery
suites** → revert commit 1 (lowest
probability; the cadence is what local already runs).
## Notes for PR 7 (bot suite cadence)
The bot suite (`single-node/bot/bot.test.ts`) uses `setup(0, {
...PIPELINING_SETUP_OPTS, ... })`, which
sets `ethereumSlotDuration: 4` explicitly. It does not read
`DEFAULT_L1_BLOCK_TIME`, so **commit 1 does
not affect the bot suite** — its cadence lever remains its own
PIPELINING preset, unchanged here.
## Measured impact
The cadence cuts land a run-wide −6.4%: −1,600s summed across shards
over the 135 suites present in
both runs (−2,046s of improvements against +446s of regressions). The
wall-clock benefit is smaller,
since shards run in parallel across workers.
Controls confirm the deltas are real cadence effects, not run-type
artifacts. `high_tps` (deliberately
pinned at the old cadence) is unchanged (+0.1s), and the WIDE_SLOT /
explicit-preset proving suites are
flat — pipeline_prune −1.2s, blob_promotion −0.1s, long_proving_time
−1.1s, prune_when_cannot_build
−0.1s. Only the default-cadence and reorg/block-production timings
moved.
Largest per-suite wall cuts (default-cadence proving/recovery from
commit 1; reorg from commit 2):
- optimistic.parallel −399.9s (−23.5%, ×8 shards)
- proof_boundary.parallel −346.0s (−25.6%, ×5 shards)
- full −202.7s; blocks.parallel −196.3s (−28.8%, ×5 shards)
- invalidate_block.parallel −118.5s (×11 shards); sync_after_reorg
−108.3s
- data_withholding_slash −72.0s; proposal_failure_recovery.parallel
−60.1s;
cross_chain_public_message −48.2s; equivocation_recovery −46.9s;
late_prover_tx_collection −43.3s
Regressions, all on high-variance multi-node suites that are not
cadence-cut here (run-to-run noise,
not attributable to this change): multi_root +74.1s, inactivity_slash
+64.4s,
attested_invalid_proposal.parallel +32.1s, proposed_chain.parallel
+26.1s, bot +22.3s.
The metric is whole-suite wall time (sum of test durations + hooks); a
cadence change carries no new
span to attribute against, so these are run-vs-run wall deltas — the
flat pinned/preset controls above
are what make the cadence attribution defensible.
Baseline CI run 1783417125211175 (merge-base 13a53f1, full). PR CI
run 1783426164237985
(x-fast).
Fixes A-1184
(cherry picked from commit eedf4eb)
…24564) ## What / why PR 2 of the round-4 e2e speedup effort. The fees-family suites pay a `setup:bridge` span (~48s ≈ 4 production slots per process) in `FeesTest.applyFPCSetup`, bridging fee juice from L1 to the BananaFPC via the gas portal (`bridgeFromL1ToL2` = prepare-on-L1 + advance 2 blocks + a claim tx). That funding is pure setup plumbing — the fees tests snapshot the FPC's gas balance and assert deltas, they don't assert on the bridge itself. Fee-juice balances are just public-data leaves at `computeFeePayerBalanceLeafSlot(address)`, which `getGenesisValues` (`world-state/src/testing.ts`) already prefills for genesis-funded accounts — the same mechanism that funds the initial accounts and (already, on this base) the sponsored FPC. This PR seeds the BananaFPC's fee juice at genesis instead of bridging it during setup. Note: the plan's premise that `fundSponsoredFPC: true` bridges was stale — the sponsored FPC has been genesis-funded since #19532 (setup pushes its deterministic address into the genesis-funded list). The remaining `setup:bridge` cost in the fees family is the BananaFPC bridge, which this PR removes. ## Mechanism - The BananaFPC address depends on the BananaCoin address and the FPC admin (the first setup account), and the accounts are generated *inside* `setup()` with random secrets — so the FPC address can't be precomputed by the test before genesis. Fixed deploy salts are added for BananaCoin and BananaFPC so both addresses become deterministic once the admin is known, and a `computeExtraGenesisFundedAddresses(defaultAccounts)` hook on `SetupOptions` runs after the accounts are generated and before genesis values are computed. `FeesTest` uses it to derive the BananaFPC address from the first account and add it to the genesis-funded list. - The returned address flows through the existing initial-accounts path (`addressesToFund` → `getGenesisValues`), so it is funded with the same fee juice as an initial account (`10^22`) and automatically included in the L1 `FeeJuicePortal` `feeJuicePortalInitialBalance` (`fundingNeeded`) accounting — the portal's locked L1 balance stays consistent with total L2 fee-juice supply, exactly as for every other genesis-funded balance. - `applyFPCSetup` deploys the FPC with the fixed salt and asserts the deployed address equals the seeded one, so any drift in the deterministic deploy params surfaces as a clear error instead of a downstream "insufficient fee payer balance". The `bridgeFromL1ToL2` call is removed. - The `setup:bridge` span wrapper is kept in place for the bridging that remains, so its disappearance from the fees setup hooks is the measured proof. ## Scope decisions (site by site) - **`FeesTest.applyFPCSetup` BananaFPC funding** → moved to genesis. Used by `account_init`, `private_payments.parallel`, `failures`, `gas_estimation.parallel`. This is the only funding that moved. - **`FeesTest.mintAndBridgeFeeJuice`** (bridges to an arbitrary recipient) → kept. It fires in `account_init` test bodies that test the bridge/claim flow ("pays natively in the Fee Juice after Alice bridges funds") — real behavior under test. - **`account_init` `FeeJuicePaymentMethodWithClaim` / `prepareTokensOnL1`** → kept. Tests the atomic claim-in-deploy flow. - **Sponsored FPC** → already genesis-funded via `fundSponsoredFPC: true` on this base; no change. - **`cross_chain_messaging_test`** (`fundSponsoredFPC: true`) → already genesis-funded; its own cross-chain bridging harness is the behavior under test, kept. - **`bench/client_flows_benchmark`** bridges to its BananaFPC → out of scope (benchmark harness, not in the fees e2e timing family), kept. - Production/sandbox funding paths (cli, `aztec sandbox`) → untouched. The new hook defaults to unset; only the e2e fixtures opt in. ## Expected effect ~48s × the number of fees processes that ran the FPC bridge in setup (account_init, private_payments, failures, gas_estimation). No C++, no new config, no change to production genesis roots. ## Local verification - `single-node/fees/account_init -t 'pays privately through an FPC'` passes; the span leaderboard for the run shows **zero `setup:bridge`** occurrences (was ~48s), while `deploy:fpc`/`deploy:token` remain. The FPC-paying test asserts the FPC's gas balance decreases by the fee, so the genesis funding is exercised end to end, and the deployed-address assertion held. - `single-node/fees/account_init -t 'after Alice bridges funds'` passes with `setup:bridge` present in the test body, confirming the retained bridge/claim harness still works. ## Measured impact The `setup:bridge` span is eliminated across the run: 17 occurrences totalling 646.4s of bridge setup time in the baseline drop to zero in the PR run. The 13 fees-hook bridges (~48s each) are the bulk of it. - Fees beforeHooks per shard: `private_payments.parallel` −51.1s (×8 shards), `gas_estimation.parallel` −48.1s (×3), `failures` −47.8s, `account_init` −47.8s — each matching the ~48s (4-slot) bridge cost. - `setup:bridge` busyMs removed per suite: account_init 48.1s, failures 48.2s, gas_estimation ×3 = 144.6s, private_payments ×8 = 384.7s (13 fees-hook bridges ≈ 625.5s), plus the ~5s sponsored-FPC bridges on `amm` and `storage_proof`. - Total beforeHooks reduction over the suites present in both runs: −907.4s summed across shards (wall-clock benefit is smaller, since the fees shards run in parallel). - One suite moved the wrong way within noise: `fee_juice_payments` +11.0s. It keeps its real bridging/claim txs (no `setup:bridge` span removed), so this is run-to-run variance, not a regression from this change. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783375878950264 (x-fast — the comparison is per-suite over the suites present in both runs). Fixes A-1405 (cherry picked from commit 1321e98)
Parallelizes the independent setup steps in the bot factory (`@aztec/bot`, `bot/src/factory.ts`). This is production bot code that runs against live networks, so the changes are strictly behavior-preserving: same contracts at the same (salt-derived) addresses, same amounts, same final state. Only the ordering/concurrency of provably-independent setup steps changes. The bot factory is the single largest cost in the `single-node/bot` e2e suite: each `*.create` runs the production sequencer at 12s L2 slots (`PIPELINING_SETUP_OPTS`) and deploys/mints one tx per slot, fully serial. `AmmBot.create` alone was ~74s (7 serial txs). ## Dependency graph per bot type Edges below are "must be mined before", verified against the Noir contract source (`token_contract`, `amm_contract`). ### Transaction bot -- `setup()` (unchanged) `setupAccount` -> register recipient -> `ensureFeeJuiceBalance` -> deploy token -> mint. This chain is fully data-dependent: funding gates the deploy, the deploy gates the mint. The recipient (`createSchnorrAccount`) is register-only (no tx), and the private+public mints are already batched into one tx. Nothing is independent, so `setup()` is left serial. ### AMM bot -- `setupAmm()` Steps: deploy token0 (D0), token1 (D1), LP token (DL), AMM (DA); `set_minter` granting the AMM rights over the LP token (SM); mint token0+token1 to the provider (M, one batched tx); `add_liquidity` (AL). - D0, D1, DL: independent -- distinct contracts, no cross-references. - DA: the AMM constructor only stores the three token addresses (no calls into the tokens). Those addresses are salt-derived, so DA needs only the (pre-derived) addresses, not the token deploys mined. - SM: `Token::set_minter` just writes `minters[amm] = true`; it does not call or validate the AMM contract. It needs the LP token deployed and the (derivable) AMM address only -- not the AMM deployed. The deployer is authorized because the Token constructor sets `minters[admin] = true`. - M: `mint_to_private`/`mint_to_public` require the caller be a minter; the deployer is admin=minter from the constructor, so the token0/token1 mints need only those tokens deployed (no `set_minter`). - AL: `add_liquidity` transfers token0/token1 from the provider (needs M for balances), mints LP tokens (needs SM so the AMM is an LP minter), and targets the AMM (needs DA). Restructured from 7 serial txs into 3 slot-phases: - Phase 1: `Promise.all([D0, D1, DL])` - Phase 2: `Promise.all([DA, SM, M])` - Phase 3: `AL` ### Cross-chain bot -- `setupCrossChain()` Steps: deploy TestContract (an L2 tx), seed N L1->L2 messages (L1 txs), wait for the first message ready. - The seeds reference only the L2 recipient (TestContract) address, which is salt-derived. L1->L2 messages are queued on L1 and do not require the L2 contract to exist yet; they are consumed later in `bot.run()`, after setup completes. So the L2 deploy and the L1 seeding are independent. - The L2 deploy pays via the L2 wallet/PXE account; the seeds use the bot's L1 client -- different accounts, so there is no L1 nonce interaction between them. Restructured to overlap the deploy with the seed loop: `Promise.all([deploy TestContract, seed loop])`, then the message-ready wait. ## What stayed serial, and why - The whole transaction-bot `setup()` (data-dependent chain, nothing independent). - `ensureFeeJuiceBalance` before every deploy (funding must precede spending). - `add_liquidity` after its mints + minter grant + AMM deploy. - The individual L1->L2 seeds inside the loop: they share the bot's single L1 account, so concurrent sends would race on the L1 nonce. The suite already documents this nonce hazard. Only the loop as a whole overlaps the (L2) TestContract deploy. ## Production safety / idempotent re-entry - Every deploy still goes through `registerOrDeployContract`, which checks `getContractMetadata(address).isContractPublished` per contract and only registers (no tx) if already deployed. If one parallel deploy fails and the others succeed, the next `*.create` recovers: the succeeded contracts register-only, the failed one redeploys. Addresses are salt-derived and identical across runs. - `set_minter` is idempotent (writes `true` again). The AMM path's mint and `add_liquidity` always run (no balance guard) -- unchanged from the previous `fundAmm`. - Same-sender concurrent `.send()`s are safe by construction: the PXE serializes simulate/prove through its `SerialQueue` and setup txs pay with random tx nonces. The bot runs the same `EmbeddedWallet` -> PXE path in the e2e suite and in production. - Deliberate, benign failure-path change: because Phase 2 runs DA/SM/M concurrently, a failure of one no longer prevents the others from being sent. Their effects (a deployed-but-unused AMM, a minter grant, an extra mint) are all idempotent-safe, and `add_liquidity` uses fixed amounts, so re-entry converges to the same final state. - `withNoMinTxsPerBlock` (the `flushSetupTransactions` save/zero/restore wrapper around each setup tx) was not safe to re-enter concurrently: interleaved save/zero/restore could read the already-zeroed value and "restore" `minTxsPerBlock` to 0 permanently. It now reference-counts entrants -- the first saves and zeroes, the last restores -- with unit tests covering the overlapping and failure paths (`bot/src/factory.test.ts`). Serial callers see the exact same RPC sequence as before. Final state is identical in every path: same contract addresses, same minter grant, same minted amounts, same liquidity. ## Local evidence Two full local runs of `single-node/bot/bot.test.ts` (production sequencer, 12s L2 slots), all 10 tests passing in both. Hook durations from factory log timestamps, against the round-4 local baseline measured on the same machine at the same cadence (findings from #24534): - `Bot.create` (transaction-bot hook, untouched): 31.8s vs 32.2s baseline -- unchanged, as intended. - `AmmBot.create`: 57.2s vs 73.9s baseline (-16.7s). The three token deploy txs go out within 300ms of each other; the AMM deploy, `set_minter` and the mint batch all go out within the following slot. - `CrossChainBot.create`: 24.0s vs 43.5s baseline (-19.5s). The TestContract deploy overlaps both L1 seeds; the first message is ready almost immediately after the deploy is mined. - Suite-level `beforeHooksMs`: 116.9s vs 153.7s baseline (-36.8s), matching the per-hook deltas. - Unit: `bot/src/factory.test.ts` (4 tests) covering the `withNoMinTxsPerBlock` reentrancy fix. ## Expected effect on CI - `AmmBot.create`: 7 serial txs -> 3 slot-phases (~87s on CI per #24534's spans -> ~55-60s). - `CrossChainBot.create`: deploy overlaps the seed pipeline (~43s -> ~25s). - Transaction bot: unchanged. - Aggregate: roughly -35 to -45s on the bot suite's beforeAll wall time. - Fix-phase proof metric (once #24534's `setup:bot` instrumentation is on this base): the amm-bot and cross-chain `setup:bot` occurrences drop per the numbers above. Measured CI numbers to be appended when a full run lands. ## Measured impact The bot suite (a single shard) drops 216.3s → 118.3s total (**−98.0s, −45%**); beforeHooks 215.9s → 118.0s. - No `setup:bot` spans exist on this base (that instrumentation lives in the unmerged #24534), so the metric is the suite-level hook fold, which covers all three nested bot factory setups (Bot, AmmBot, CrossChainBot). - Noise context from the same run pair: untouched light suites move −2 to −3s per shard (private_initialization −2.9s/shard over 20 shards, deploy_method −2.1s/shard over 14), and the largest counter-move is validators_sentinel +24s (multi-node variance). A −98s single-shard delta is far outside that envelope. - The CI delta exceeds the local −36.8s because the CI baseline pays more missed-slot penalties per serial tx (baseline beforeHooks 215.9s on CI vs 153.7s locally at the same 12s cadence); collapsing 7 serial txs into 3 slot-phases removes proportionally more where slots are more often missed. Baseline CI run 1783393028773172 (merge-base 30966a4, full). PR CI run 1783427250198215 (x-fast). Fixes A-1408 (cherry picked from commit 5d2b6df)
Round-4 e2e speedup, PR 1b (adoption half of the AuthRegistry prize). **Stacked on #24567** (`spl/genesis-prefilled-nullifiers`, the genesis-nullifier mechanism). This PR targets that branch; retarget to `merge-train/spartan-v5` once #24567 merges. ## What Make e2e environments start with the standard contracts (AuthRegistry, PublicChecks, HandshakeRegistry) already "published", so `ensureAuthRegistryPublished` and its two siblings skip their two publish txs each. At production cadence `setup:auth-registry` alone was ~16 min/run (2 sequential txs per process, paid per-process across the suite). This collapses that span to sub-second. ## Mechanism (three parts, all default-off outside e2e) - **Genesis nullifiers.** Per standard contract, seed `siloNullifier(ContractClassRegistry, classId)` and `siloNullifier(ContractInstanceRegistry, instanceAddress)` (the real derived address — standard contracts are not magic-address protocol contracts) into the fixtures' `getGenesisValues` call, via PR 1a's new 5th param. These are exactly the nullifiers the publish txs would emit, so the AVM's deployment-nullifier check passes when the contracts are called publicly. A single helper (`getStandardContractGenesisNullifiers`) feeds all four e2e genesis builders (`fixtures/setup.ts`, `e2e_prover_test.ts`, `p2p/p2p_network.ts`, `multi-node/governance/add_rollup.test.ts`) — the latter three recompute a genesis that must reproduce the L1-deployed archive root, so they must seed the identical set. Non-e2e callers (cli, sandbox/local-network) are untouched. - **Archiver preload.** `registerStandardContracts` mirrors `registerProtocolContracts`: it seeds each contract's class (with recomputed `publicBytecodeCommitment`), instance, and public-function signatures into the contract store at block 0, reading the bundled `@aztec/standard-contracts` artifacts. It is idempotent (skips already-registered classes on restart) and gated behind a new archiver config flag `testPreloadStandardContracts` (env `TEST_PRELOAD_STANDARD_CONTRACTS`, default **false**). The flag is set from the e2e node config so every spawned node (validators, prover nodes) picks it up through the normal config path. This adds `@aztec/standard-contracts` as an archiver dependency. - **Guard short-circuit.** The `ensure*Published` helpers are unchanged. Their guards read `wallet.getContractClassMetadata(id).isContractClassPubliclyRegistered` (to `aztecNode.getContractClass`) and `wallet.getContractMetadata(addr).isContractPublished` (to `aztecNode.getContract`) — both are archiver-store reads, not nullifier-tree reads. The store preload alone makes both short-circuit; the genesis nullifiers are what make the contract actually callable in the AVM afterwards. Keeping the helpers doubles as a regression check: if seeding ever breaks, the tx path re-engages and tests still pass, just slow. ## Why the flag is test-only (A-1257 rationale) Preloading unconditionally in production would recreate the A-1257 collision #24254 fixed: a real on-chain publish of a preloaded class would collide with the block-0 preload, because production genesis will **not** carry the matching nullifiers. The flag is only set by the e2e fixtures, which also seed the nullifiers, keeping the store and the nullifier tree consistent. A single source of truth (`getPublishableStandardContracts`) drives both the preloaded set and the seeded-nullifier set so they cannot drift. ## State-write verification Inspected the two publish txs on this base: - `ContractClassRegistry.publish` pushes one nullifier (`classId`, siloed to the class-registry address) and broadcasts a `ContractClassPublished` contract-class log carrying the packed bytecode. No public-data writes. - `ContractInstanceRegistry.publish_for_public_execution` asserts the class-registration nullifier exists, then pushes one nullifier (`address`, siloed to the instance-registry address) and broadcasts a `ContractInstancePublished` private log. No public-data writes. So the full state-write set is two nullifiers + two broadcast logs. The nullifiers are replicated via genesis `prefilledNullifiers`; the archiver normally learns the class/instance from those two logs (`data_store_updater.ts`), and the block-0 preload replaces that path exactly. No `genesisPublicData` is required. PXE-side registration (`wallet.registerContract`) is unaffected and still runs in the helpers. ## Local verification - `automine/accounts/authwit.test.ts` (public-authwit path against the standard AuthRegistry), timing on: passes; `setup:auth-registry` span = **10 ms** (was ~33 s at production cadence). No on-chain AuthRegistry publish (the only `ContractClassRegistry.publish` executions are the test's own AuthWitTest/GenericProxy deploys); the public AuthRegistry contract executes fine against the seeded nullifier. `testPreloadStandardContracts: true` confirmed in the node config dump. - `single-node/fees/account_init.test.ts` (production sequencer + simulated prover node), timing on: 5/5 pass; `setup:auth-registry` span = **10 ms**. Prover node syncs against the seeded genesis (no root divergence), and the flag propagates to it. No duplicate-nullifier errors. - New unit test in `archiver/src/modules/data_store_updater.test.ts`: `registerStandardContracts` preloads each publishable standard contract's class + instance into the store and is idempotent on a second call. ## Expected impact ~14–16 min/run from auth-registry alone, plus more from the two untagged siblings (PublicChecks / HandshakeRegistry) wherever used. ## Measured impact `setup:auth-registry` collapses from 702.2s of setup time run-wide (39 occurrences, 5–35s each) to 482ms total (max 87ms per occurrence) — sub-second run-wide, every shard clean, no seeding or flag-propagation slow path. - The measured beforeHooks reduction is larger than the auth-registry span alone, because the two untagged sibling publishes (PublicChecks, HandshakeRegistry) are removed too. Example: account_init beforeHooks −49.6s vs a 30.9s tagged auth-registry span; the extra ~19s is the two siblings. - Per-suite beforeHooks deltas: account_init −49.6s, fee_settings −49.5s, failures −48.7s, private_payments.parallel −41.8s/shard (×8), gas_estimation.parallel −41.5s/shard (×3), l1_to_l2 −38.1s, l2_to_l1 −37.5s, l1_to_l2_inbox_drift −38.2s, token_bridge −37.7s, fee_juice_payments −37.4s, bot −34.3s; automine/parallel suites shed 5–15s each (their fast-cadence publishes). - Total beforeHooks reduction over the suites present in both runs: −1,649.7s summed across shards (wall-clock benefit is smaller, since shards run in parallel across workers). Baseline CI run 1783389062016888 (#24567 `ci/x-fast`; mechanism-only, so timing-neutral vs merge-base). PR CI run 1783391194852197 (#24568 `ci/x-fast`). ## Notes for downstream - **PR 6 (fees harness):** `applyEnsureAuthRegistryPublished` now short-circuits; the fees setup chain loses the ~32 s/shard auth-registry step, so rebase the overlap/batch shape onto what remains (token deploy, FPC, mints). - **Fix phase / timing capture:** expected span assertion is `setup:auth-registry` around sub-second across the run (10 ms observed locally per process). Any shard still showing tens of seconds means the guards took the slow path (a seeding or flag-propagation bug), not noise. Fixes A-1404 (cherry picked from commit f320049)
…state (#24578) v5 version of #24558 (same change, based on `merge-train/spartan-v5`). ## Problem When the prover broker cancelled a proving job it settled the job as a cached `{ status: 'rejected', reason: 'Aborted' }` result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (`Cached proving job … Not enqueuing again`) and the facade failed the job with `Aborted`. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune. ## Fix Cancellation is now its own first-class, **revivable** state: - A new `aborted` proving-job status (`ProvingJobStatus` / `ProvingJobSettledResult`). `cancelProvingJob` records it, notifies the current waiter, and **persists** it (`ProvingBrokerDatabase.setProvingJobAborted`). - `enqueueProvingJob` **revives** an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the new `deleteProvingJobResult`, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminal `rejected` results. Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving. The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place. ## Tests `proving_broker.test.ts` (both DB variants): cancel leaves a job `aborted`; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted). Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit a4e88ca)
Adds `yarn-project/.claude/skills/writing-e2e-tests/SKILL.md`: a skill
that gives an agent everything it needs to place, structure, and write a
robust e2e test with little guidance. Written against the v5-next e2e
layout (category directories with per-category READMEs and context
classes).
## What it covers
- **Step 0 decision ladder**: unit test → new expectation in an existing
test → new `it` in an existing suite → new file on an existing
context/harness → new standalone test.
- **Where to place the test**: recap of the category directories
(`automine/` via `AutomineTestContext`, `single-node/` via
`setupWithProver`/`setupBlockProducer`, `multi-node/` via
`MultiNodeTestContext` on the mock-gossip bus, `p2p/` via
`P2PNetworkTest` on real libp2p, plus `composed/`, `infra/`, `spartan/`,
`bench/`), the multi-node-vs-p2p decision rule, pointers to the
per-category READMEs as the authoritative reference, file naming
(`describe` matches path, header comment), automatic CI registration
through `bootstrap.sh` globs, `.parallel.test.ts` semantics, and
jest/bash timeout sync.
- **Setup reuse**: the three layers (category contexts → domain
harnesses → root `SetupOptions`), the standard suite file shape,
one-environment-per-file, preset spread order, guarded teardown.
- **Readability**: intent-only test bodies, the named-waiter surface
(`fixtures/wait_helpers.ts`, context waiters, `ChainMonitor`), shared
helpers and co-located `*_test_helpers.ts`, simulators with `afterEach`
checks, shared error constants.
- **Speed**: themes distilled from the recent
`test(e2e)`/`perf(e2e)`/`chore(e2e)` speedup PRs and their tracking
issues — avoid unneeded setup, genesis seeding over setup txs,
`BatchCall` batching over `Promise.all` overlap (with the
PXE-serialization caveat), warping dead waits with
`markProvenAndWarp`/`warpWithSequencersPaused` ("an honest wait beats a
flaky warp"), named timing profiles over ad-hoc cadences, and measuring
with the `testSpan` instrumentation + `track-e2e-times` skill before
optimizing.
- **Flakiness**: twelve golden rules synthesized from six months of
deflake PRs and the accumulated flaky-test gotchas (poll-don't-sleep via
named waiters, receipt-anchored assertions, mined ≠ checkpointed ≠
proven, `syncChainTip` tag awareness, pipelining timing margins +
`findSlotsWithProposers`, invariant-not-exact-value assertions, L1
account/nonce and port hygiene, freeze-L1-across-restarts, gossip-mesh
readiness via `runGossipScenario`, fee padding, determinism, honest
timeouts), plus `deflaker.sh` validation and `.test_patterns.yml` as a
last resort.
- Final pre-ship checklist.
All helper/API names cited were verified to exist on
`merge-train/spartan-v5` (`AutomineTestContext.setup`,
`setupWithProver`/`setupBlockProducer`,
`MOCK_GOSSIP_MULTI_VALIDATOR_OPTS`, `wait_helpers.ts` waiters,
`warpWithSequencersPaused`, `markProvenAndWarp`,
`testSpan`/`TEST_TIMING_FILE`, `runGossipScenario`,
`waitForP2PMeshConnectivity`, `findSlotsWithProposers`,
`getPaddedMaxFeesPerGas`, etc.). Genesis-prefill techniques that only
exist in still-open speedup PRs are described as themes to look for
rather than as concrete APIs.
(cherry picked from commit 4c2b6d8)
Stacked on #24564; retarget to `merge-train/spartan-v5` after it merges. (Unifies former #24584 into this PR — the two touched the same mint path in `fees_test.ts`, so they now land together.) Restructures the serial setup-tx chains in the fees and cross-chain e2e harnesses so independent setup transactions stop paying one production slot each: independent txs run concurrently, and same-sender calls are batched into a single tx (one proof, one tx, one slot). Part of the round-4/5 e2e speedup effort. ## Commits - **perf(e2e): overlap fees harness setup txs** — after `FeesTest.setup` deploys BananaCoin, the BananaFPC deploy, the SponsoredFPC registration (a PXE registration, no L2 tx), and Alice's banana mints each depend only on the token, so the fees suites (`account_init`, `gas_estimation`, `private_payments`) now run them under a `Promise.all`. The cross-chain harness (`deployAndInitializeTokenAndBridgeContracts`) deploys the L2 token and its bridge concurrently: the bridge takes the token address as a constructor arg, but that address is deterministic before the deploy tx mines (the deploy is now constructed with an explicit `deployer`, same address derivation as the previous send-time lock), and the bridge constructor only stores it without calling into the token. - **perf(e2e): parallelize dual token mints** + **perf(e2e): batch same-sender setup txs into single txs** — `applyFundAliceWithBananas` sent Alice's private and public banana mints back-to-back (measured 100% serial: `tx:mint` busyMs == totalMs on the base). The first commit made them concurrent; the second replaces the concurrency with a single `BatchCall([mint_to_private, mint_to_public])` tx, because measurement showed concurrent same-sender txs still pay *consecutive* slots: the PXE serializes `simulateTx`/`proveTx` on a single queue, so the second tx is only proven after the first and misses the first's slot. Overlapping cannot remove whole slots — only batching can. Both mints are Alice sends on `BananaCoin` touching disjoint balances; the private-balance before/after assertion is preserved, and the public mint's effect is checked by the downstream tests that spend Alice's public bananas. Precedent for the pattern already lives in this tree (`mintNotes` in `fixtures/token_utils.ts`). ## Dependency analysis - FPC deploy needs the token address (constructor arg) → runs after the token deploy, concurrent with the mint batch. - Mints need the token → concurrent with the FPC deploy, and the private/public mints are mutually independent (disjoint balances), so they share one tx. - SponsoredFPC setup is a wallet registration only → free to join the `Promise.all`. - TokenBridge's initializer stores `{token, portal}` in its own storage and never calls the token, so the bridge deploy does not need the token to be mined first. - #24564 invariants preserved: `BANANA_COIN_SALT`/`BANANA_FPC_SALT` + `deployer: alice` are untouched, and the genesis-funded-address assertion in `applyFPCSetup` still runs (and passed locally). ## Same-sender concurrency Concurrent `.send()` calls from the same sender through the same TestWallet are safe: the PXE serializes `simulateTx`/`proveTx` through a `SerialQueue` (`#putInJobQueue`), so simulations never interleave and only the block-inclusion waits overlap. The setup txs pay fees from Alice's public fee-juice balance (`PREEXISTING_FEE_JUICE`) with random tx nonces, so there are no nullifier or note conflicts between the concurrent txs. Verified locally with full `Promise.all` of the send+wait calls; no NO_WAIT fallback or cross-sender split was needed. ## Surveyed and skipped - Batching token class+instance publication: not applicable on this base — `DeployMethod.request()` already merges class publication, instance publication, and the constructor call into a single execution payload sent as one tx. Measured locally, `deploy:token` is a single ~12s (1 slot) occurrence; the ~24s occurrences in CI span data are inclusion latency at CI cadence, not a second tx. - `applyFPCSetup`: a single `deploy:fpc` tx — no same-sender pair inside the phase. Folding the FPC deploy into the mint batch (one class publication + app calls is within the class-log limit) would remove the remaining consecutive slot, but it merges two harness phases that four test files compose differently via `Promise.all`, and deploy-inside-BatchCall is not a proven pattern here — a restructure, not a mechanical merge. Left as a possible follow-up. - `applySponsoredFPCSetup`: no on-chain tx — it only calls `wallet.registerContract` locally; the sponsored FPC is genesis-funded and its class preloaded. Nothing to batch. - Two different contract deploys cannot share a tx (`MAX_CONTRACT_CLASS_LOGS_PER_TX = 1`), which rules out any two-deploy batch (e.g. cross-chain token+bridge) — those stay concurrent rather than batched. ## Local verification (12s slots; CI runs at its own cadence) Overlap commits (fees + cross-chain): - Base (`account_init`, spans on): 5/5 passed; beforeAll 83.5s = `setup:auth-registry` 32.0s + `deploy:token` 12.0s + `tx:mint` 23.9s (busyMs == totalMs, fully serial) + `deploy:fpc` 11.8s. - After overlap: 5/5 passed; beforeAll 72.2s (−11.3s); `deploy:fpc` fully overlapped with the mint window. - `gas_estimation`: 3/3 passed; `private_payments`: 8/8 passed; `token_bridge` (cross-chain overlap): 7/7 passed — `deploy:token` 12.1s and `deploy:bridge` 23.7s ran concurrently (union ≈ 23.7s vs ~48s serial on the CI baseline); `l2_to_l1`: 6/6 passed (regression check on the untouched path). Batch commit (`account_init`, 5/5 green on all runs): | run | tx:mint count | tx:mint busyMs | deploy:fpc busyMs | beforeHooksMs | |-----|---------------|----------------|-------------------|---------------| | before batch | 2 | 23618 | 23713 | 71607 | | run 1 (after) | 1 | 11834 | 24065 | 72372 | | run 2 (after) | 1 | 11807 | 23743 | 71678 | The mint phase drops from two consecutive slots (~23.6s) to one (~11.8s), one proof and one tx fewer per shard. No intermittent failures across any run. ## Measured impact Measured on CI in two steps (each commit group had its own PR run before unification). **Overlap** (PR run 1783394912998666 vs #24564's run 1783375878950264): −162s summed across shards. Per-suite beforeHooks: private_payments.parallel −16.0s (×8), gas_estimation.parallel −15.5s (×3), token_bridge −12.0s, fee_juice_payments −12.0s, account_init −11.5s. The overlap is real but partial: `tx:mint` busyMs/totalMs = 0.68 run-wide — concurrent txs land in consecutive slots because the PXE serializes proving (the finding that motivated the batch commit). Cross-chain `token_bridge` collapses its deploy phase from serial 35.9s to concurrent ~23.4s (union ≈ max, as designed). **Batch** (PR run 1783432285938536 vs 1783394912998666): `tx:mint` 42 → 29 occurrences (one merged per fees shard); busyMs 523s → 431s (−92s of tx/proving work; busyMs == totalMs in the PR run — no overlapping mint txs left to union). Wall-clock lands where the mint is the setup critical path: `failures` −12.0s, `fee_juice_payments` −11.5s (≈ one 12s slot each). Suites where the mint overlaps `applyFPCSetup` are flat (`account_init` −0.4s, `private_payments.parallel` +2s, `gas_estimation.parallel` +3s): `deploy:fpc` is unchanged (13 occurrences, ~2 slots each) and remains their critical path, so removing a slot from the mint branch of the `Promise.all` doesn't move the union. Run-wide sum over common suites: −8s; noise envelope on untouched suites ±20–37s. Net vs #24564's baseline: roughly −185s summed across shards, plus 13 fewer txs/proofs per run (less proving contention through the sequencer). The remaining ~12s/shard lever on the overlapped fees suites is folding the FPC deploy into the same batch, noted above as a follow-up. Fixes A-1406 Fixes A-1409 (cherry picked from commit 1d280af)
Removes `yarn-project/THREAT_MODEL.md`, added in [#24465](#24465). Per discussion in #team-alpha: detailed threat models and invariants may make it easier for attackers to find bugs, so the team wants this content pulled from the public repo for now. It will be re-added to LabsBox/ClaudeBox (private/internal) — see [claudebox#1357](AztecProtocol/claudebox#1357) — and can come back here once the team is finding fewer bugs. Only the threat model doc itself is removed here — the incidental doc cleanups bundled into the same squashed commit (stale BLOCK protocol references in `yarn-project/p2p/README.md` and `yarn-project/p2p/src/services/reqresp/README.md`, and the missing `DUPLICATE_ATTESTATION` entry in `yarn-project/slasher/README.md`) are kept as-is. (cherry picked from commit 4d02692)
Backport of #24512 ("docs: document initializerless accounts") into `v5-next`, requested by Alejo in Slack. ## What this adds (same as the original PR) - **New concept page** `foundational-topics/accounts/deployment.md` ("Account Deployment"): the standard initialize-and-deploy flow vs the initializerless flow, how the signing key is committed into the address via `immutables_hash`, and the trade-offs (no deploy cost/delay; signing key fixed forever; per-PXE local setup; `getDeployMethod()` throws). - **New "Create an initializerless account" section** in `aztec-js/how_to_create_account.md` with the `createSchnorrInitializerlessAccount` snippet and caveats. - **Cross-links** from `fees.md`, `wallets.md`, `contract_creation.md`, and `aztec-nr/framework-description/immutables.md`. ## How the backport differs from the original - The original PR mirrored all changes into `developer_versioned_docs/version-v5.0.0-rc.2/`. That snapshot does not exist on `v5-next` (the only snapshot here is `version-v4.3.0`, which documents the v4 line and must not describe a v5-only feature), so this backport touches **only the current docs** under `docs/docs-developers/`. Git's directory-rename detection tried to remap the mirrored copies into the v4.3.0 snapshot during the cherry-pick; those were dropped deliberately. - Everything else is the original commit cherry-picked verbatim; the only diff-vs-original differences are context lines where `v5-next` prose already diverged from `next`. ## Notes - Unlike `next`, `createSchnorrInitializerlessAccount` **does** exist in this branch's `yarn-project` (verified), so the fenced snippet could be converted to an `#include_code` example here as a follow-up. Kept as a fenced block for a faithful backport. - Verified all new internal links resolve to files present on `v5-next`, and `initializerless` is already in `docs-words.txt` on this branch. Refs #24512. --- *Created by [claudebox](https://claudebox.work/v2/sessions/045227c21dcc966b) · group: `slackbot`* (cherry picked from commit 7534604)
While trying to deploy the latest nightly standard contracts to testnet I experienced gas estimation issues. According to Claude these issues are due to variability in Merkle tree insertions: there's around 40% of gas cost spread between the happiest case (no tree "climbing" needed) and the worst case (10 level climb). If eth_estimateGas occurs during the happiest case, and actual TX execution during the worst case, the default 20% buffer we use in Aztec.js turns out not to be sufficient. Closes F-794 (cherry picked from commit 2734da1)
(cherry picked from commit 23e8e1e)
waitForNode called retryUntil without a timeout, and retryUntil treats a
zero timeout as the never-time-out sentinel, so an unreachable node caused
the call to poll getNodeInfo forever with no way for the caller to bound it.
Add an optional { timeout, interval } argument and default it to
DefaultWaitOpts (300s / 1s), matching the sibling waitForTx. An unreachable
node now rejects with a TimeoutError; pass timeout: 0 to opt into the
previous infinite-wait behavior.
(cherry picked from commit 482a6b8)
First part of integrating the enhancements explored at aztec-labs-eng/oxide#257 Closes F-808 (cherry picked from commit 8a3f7e6)
…poch (#24436) Resolves A-1290. ## Problem A `CheckpointProver`'s sub-tree work forks world-state per block. When an L1 reorg prunes a base block, those fork reads fault and the prover permanently rejects its one-shot `blockProofs` promise. The store keeps the prover alive across a prune (`markPruned`) so a re-add of identical content can reuse its in-flight work. But a prune that removes the checkpoint also breaks the fork reads, so that reuse only ever hands back a poisoned prover: a re-add or a full `EpochSession` recreate re-references the same rejected `blockProofs` and fails immediately. The affected node then silently abandons the epoch until a process restart (a healthy prover elsewhere still proves it, so it's not a network stall). ## Fix **Drop the reuse machinery — a pruned prover cannot survive, so rebuild instead of reuse.** - `CheckpointStore.cancelAndRemoveAboveBlock` cancels and **removes** orphaned provers (replacing `markPrunedAboveBlock`); a re-add builds a fresh prover with a fresh `blockProofs`. - Removed the `pruned` flag / `markPruned` / `markCanonical` from `CheckpointProver`, and the `SlotWatcher` that only reaped lingering pruned provers. - `listCanonical` / `addOrUpdate` simplified — every prover in the store is now canonical. **Recover the epoch by rebuilding on re-add, rather than trying to prevent the terminal state.** A prune unwinds world-state as soon as it's detected — before the prover-node cancels the affected session — so a checkpoint prover mid-fork faults while still live and its session goes terminal `failed`. That race is inherent (the world-state rejection and the prover being cancelled are unordered), so instead of trying to reclassify it as a cancellation, we make the terminal state harmless: - `SessionManager.openFullSessionIfReady` no longer treats a terminal session as "already open" — it drops it and rebuilds. So re-adding an epoch's checkpoints reopens a fresh live session over fresh provers, via the checkpoint/prune triggers (ungated by the tick high-water mark). This reopen depends on the store no longer holding the poisoned prover: with the reuse machinery gone the rebuilt session gets a fresh prover with a healthy `blockProofs`, so it can actually prove rather than re-inheriting the rejected one. - `runSession` skips the failure-upload when the session's checkpoints no longer match canonical content — a prune-invalidated failure isn't a genuine proving failure, so it no longer emits a spurious post-mortem upload / alert. The only non-recovered case is "content pruned and never re-added" (chain stops publishing that epoch's checkpoints) — there's nothing to prove on this node then, so abandoning the epoch is correct. **Cancel the pruned prover's in-flight work promptly.** `CheckpointProver` now threads its abort signal into `PublicProcessor.process`, so a prune-driven cancel stops the current block's public execution immediately instead of running it to completion before the next `signal.aborted` check. This is independent of the recovery logic above — it just avoids wasting CPU re-executing a block whose checkpoint is being discarded. The in-flight block has not yet enqueued a broker proof, so aborting it loses no reusable work. ### Note: broker-level proof reuse is unaffected Removing the `CheckpointProver`-level reuse does not waste proving work. The expensive part — the SNARK proofs — is cached in the proving broker, content-addressed by job id, independent of world-state. `cancelJobsOnStop` defaults to `false`, so cancelling a prover does not abort its broker jobs; on an identical re-add the fresh prover regenerates identical inputs, and the broker dedups against the cached jobs/results (bounded by the broker's per-epoch cleanup). The reuse we deleted was witness-generation reuse, whose world-state forks cannot survive a prune anyway. ## Testing - `session-manager.test.ts`: pins the retry/recovery invariant — the periodic tick does not re-attempt a failed epoch (gated by `lastTickEpoch`), but a checkpoint re-add recovers it (ungated); a `failed` session's epoch is reopened once its checkpoints are re-added; a genuine proving failure still uploads a post-mortem; a failure coinciding with a canonical content change (prune) skips the upload. (Red/green verified for both the gate and the upload suppression.) - `checkpoint-store.test.ts` / `prover-node.test.ts`: pruned provers are cancelled and removed (not flagged); a re-add builds a fresh prover. - `checkpoint-prover.test.ts`: cancelling a prover mid-block aborts the signal its public execution is running under (red/green verified). - Full prover-node suite passes (166 tests); package typechecks clean; lint OK. - Updated `optimistic.parallel.test.ts` (the e2e that spawned this issue) to the cancelled+removed semantics. ## Notes - No operator/contract-dev-facing config or API changed, so no changelog entry. - The terminal `failed` state is not prevented, only recovered from: the world-state unwind and the prover cancellation are unordered, so the failure can't be reliably reclassified from the session side. Recovering on re-add is deterministic and needs no such race to be won. (cherry picked from commit c1e6229)
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
Additions to the README on gas and fees (cherry picked from commit 4d53a9f)
Some stores were being accessed without checking if the associated accounts were in scope or not. For logs I added this to the service, but we're not yet consistent in how we use services (call-lived, tx-lived?) so in other cases it is just inlined in the oracle. (cherry picked from commit ce12e29)
I added a sanity check to avoid having messages implausibly far into the future, which would lead to them not getting evicted etc., but more importantly which would signal some inconsistency somewhere. (cherry picked from commit fdc2ac5)
…24665) ## Problem When encrypting a message, the ephemeral secret key comes from an unconstrained routine, so a malicious sender can substitute any value while proving. Substituting `eph_sk = 0` yields the point at infinity as the ephemeral public key, which passed the y-sign check: its y-coordinate is 0, which counts as positive. Its x-coordinate (0) is then broadcast, but 0 is not a valid x-coordinate on the curve, so the recipient can never reconstruct the key and the message is permanently undecryptable. This breaks the constrained-delivery guarantee that a note delivered by an untrusted sender remains decryptable by the recipient. ## Fix `generate_positive_ephemeral_key_pair` now asserts the ephemeral public key is not the point at infinity, alongside the existing sign check. A test emulates the substitution by mocking the randomness oracle to return 0. Fixes F-799 (cherry picked from commit 47c30a1)
Upstreaming the utilities build for `aztec-kit` so external projects can benefit from our e2e scaffolding. Essentially the same thing we already use, but pure ts rather than the unpublishable .sh script. Verified with `ci-full-no-test-cache` (cherry picked from commit 7e09008)
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
…evidence (#24655) Fixes the sender tag sync conflict from https://gist.github.com/nventuro/0aa690736b1d2865e27197723a814e9d, including the "Window straddle" residual edge. Discovery could re-derive a different index range for an already tracked (secret, txHash) pair and hit the `Conflicting range` throw in `storePendingIndexes`, permanently wedging the secret. Two triggers: - A same-PXE tx partially reverts: the chain only shows the surviving non-revertible sub-range of the prove-time entry, and the throw fired before the finalized receipt step could resolve it. - A tx straddles a sync window boundary: the window loop assembles its range piecewise, so window 2 stores a different range than window 1 (latent regardless of reverts). Fix: discovery passes `mergeExisting` to `storePendingIndexes`, which widens the stored entry to the union of both ranges (grow-only, driven by onchain evidence). The finalized receipt step still resolves partial reverts. The only other writer of pending ranges, `persistSenderTaggingIndexRangesForTx` (records the indexes a tx sent from this PXE used, at prove time), does not pass the flag and still throws on a mismatch: there it indicates a bug, not partial onchain evidence. Red-green (all fail on the base with the `Conflicting range` throw): - partial revert repro: finalizes the surviving index, frees the squashed ones, repeat sync is a no-op - cross-sync widen: an entry tracked from an earlier sync grows when discovery evidences further indexes - window straddle: drives the actual window advance loop, asserts per window queried tags and that the widened entry later finalizes cleanly Store-level tests pin the union semantics: a sub-range keeps the entry, a range beyond it widens it, an untracked tx still stores. (cherry picked from commit fd7515d)
## Problem Partial-note discovery has four crash points on the completion path, each reachable by a malicious sender and each firing before the pending note advances — so every subsequent sync re-hits it and permanently freezes note sync for that contract: - a matched completion log yields no note (`panic`), - a pending note resolves to more than one completion log (`assert`), - the delivered private half plus the log's public content exceed the packed-note capacity (`BoundedVec` overflow in the append), or - the completion log payload is empty, so reading the storage slot is out of bounds. The first three are reachable on the canonical token (mismatched content over the unconstrained delivery channel, completing the same partial note twice which the token does not prevent, and an over-length delivered private half); the last needs an attacker-controlled contract emitting a tag-only log. ## Fix Make each non-fatal: warn and advance the FSM rather than panicking, so one bad message cannot break sync. A completion log that cannot yield a note (empty, over-length, or matching none) is skipped; more than one completion log completes with the first. Fixes F-798 (cherry picked from commit 6489102)
(cherry picked from commit 3100065)
(cherry picked from commit 50dd0cb)
…l validation (#24694) ## Problem On a live mainnet node, checkpoint-proposal validation could throw an uncaught error out of the gossipsub message handler whenever it ran while the world-state synchronizer trailed the archiver (block source) by a block: ``` ERROR world-state:database Call CREATE_FORK failed: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ERROR p2p:libp2p_service Error handling gossipsub message: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ``` The archiver already had block N (so the block lookups inside checkpoint validation succeeded), but world state's `unfinalizedBlockHeight` was still N-1. When `validateCheckpointProposal` forked world state at the parent block before it had been applied, the raw tree error escaped as an uncaught ERROR-level gossipsub message. The node lost that checkpoint validation/attestation for the round and spammed error logs; it self-healed the next tick, but it should not happen. ## Root cause The two world-state fork sites were asymmetric. The block re-execution path (`reexecuteTransactions`) syncs world state *before* forking. The checkpoint validation path (`validateCheckpointProposal`) forked via `checkpointsBuilder.getFork` *without* a preceding sync. The checkpoint path does sync the block source (archiver) earlier, but that does not advance the world state, so the gap remained. ## Fix - Move the sync-before-fork invariant into `FullNodeCheckpointsBuilder.getFork`: it now calls `worldState.syncImmediate(blockNumber, blockHash)` before `worldState.fork(blockNumber)`, so the single fork helper owns the invariant rather than each caller. `syncImmediate` blocks until world state reaches the block, or throws a typed error if it genuinely cannot, instead of leaking the raw tree error. - In `validateCheckpointProposal`, look up the parent block's hash from the block source and pass it through so the sync re-syncs on a world-state reorg. Wrap the fork in a try/catch that maps any failure to a clean, non-slashable `world_state_not_synced` result, so a sync/fork failure never surfaces as an uncaught ERROR-level gossipsub message. - As a second, independent guard, check that the forked world state's archive root matches the proposal's `lastArchiveRoot` before rebuilding the checkpoint. A mismatch means world state forked from a different chain than the proposal was built on, so we fail fast with a clean, non-slashable `initial_archive_mismatch` result — mirroring the block-proposal re-execution check — instead of a confusing downstream header/archive mismatch. Both new reasons record an `unvalidated` outcome and are non-slashable, matching the other "we couldn't validate right now" / local-state-divergence reasons. Other checkpoint-builder fork sites were reviewed and need no change: the proposer's `checkpoint_proposal_job` forks at its own already-synced, coherence-checked world-state tip, and the block-proposal re-execution path already syncs before forking and performs the equivalent archive-root check. ## Tests - `checkpoint_builder.test.ts`: `getFork` syncs world state to the block before forking, and propagates a sync failure without forking. - `proposal_handler.test.ts`: forks at the parent block number passing its block hash; a fork failure returns `world_state_not_synced`; and a fork whose archive root diverges from the proposal returns `initial_archive_mismatch`. ## Notes The same bug exists on the v4 line — the v4 site is the same function, forking via `this.worldState.fork(parentBlockNumber)` directly without a preceding `syncImmediate`. This fix should be ported there as well. Fixes A-1421 (cherry picked from commit b7a7d1b)
This fixes a scoping bug where an account's ivsk would be used to produce an ECDH shared secret even if that account was not in scope. The base wallet class was adjusted to put these in scope when an explicit sender is selected for tagged messages. (cherry picked from commit 4994006)
(cherry picked from commit c6ab912)
Replace the secp256r1-specific unique/positive table_index test with one that sweeps every basic table reachable through any MultiTable. The LogDeriv relation identifies a table solely by table_index, so this guards against any generator (not just secp256r1 fixed-base) storing a window or bit-slice position in place of the builder-assigned index. (cherry picked from commit 3a53c97)
(cherry picked from commit f013e44)
PhilWindle
force-pushed
the
port-v5-backlog-to-next
branch
from
July 21, 2026 16:20
66aad56 to
0c1693a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backlog: clean forward-ports from v5-next
One-off sweep of genuinely-new
v5-nextwork merged afternextwas cut, ported per-commit. This PR contains only the commits that cherry-picked cleanly (no conflicts). Kept as individual commits for review (ci-no-squash).Deliberately not on
port-to-next-staging, so it does not auto-merge — review and merge manually.Companion to #24848 (the automation that handles this going forward). Conflicting v5 PRs are being resolved individually via the
port-to-nextlabel pipeline (see the tracking gist).Commits (35)